Conversation
There was a problem hiding this comment.
Pull request overview
This PR addresses SQL Server cross-row contention in Umbraco’s distributed locking by forcing row-level locks on the umbracoLock table, and adds integration tests intended to reproduce/validate the contention scenario described in #22113.
Changes:
- Add
ROWLOCKhint to SQL Server distributed locking SQL (NPoco + EF Core) to avoid page-lock contention onumbracoLock. - Add new SQL Server-only integration tests that simulate/force page locks vs row locks across
ContentTreeandDistributedJobslock IDs.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| tests/Umbraco.Tests.Integration/Umbraco.Infrastructure/Persistence/LocksTests.cs | Adds new long-running concurrency/locking tests to demonstrate page-lock contention and validate ROWLOCK behavior. |
| src/Umbraco.Cms.Persistence.SqlServer/Services/SqlServerDistributedLockingMechanism.cs | Updates SQL Server NPoco locking queries to include ROWLOCK. |
| src/Umbraco.Cms.Persistence.EFCore/Locking/SqlServerEFCoreDistributedLockingMechanism.cs | Updates SQL Server EF Core locking queries to include ROWLOCK. |
You can also share your feedback on Copilot code review. Take the survey.
tests/Umbraco.Tests.Integration/Umbraco.Infrastructure/Persistence/LocksTests.cs
Outdated
Show resolved
Hide resolved
tests/Umbraco.Tests.Integration/Umbraco.Infrastructure/Persistence/LocksTests.cs
Outdated
Show resolved
Hide resolved
tests/Umbraco.Tests.Integration/Umbraco.Infrastructure/Persistence/LocksTests.cs
Outdated
Show resolved
Hide resolved
88db541 to
4f53251
Compare
Migaroez
left a comment
There was a problem hiding this comment.
This seems like a sensible change and it for sure enforces the design of the data. Only the Contexts (each represented by a single entry in the table) that are relevant to an operation should be locked, the others should be left alone.
I doubt that this will make the locking issues we will most likely reduce the their occurance.
Updated [Umbraco.Cms.Persistence.Sqlite](https://github.com/umbraco/Umbraco-CMS) from 17.2.2 to 17.3.0. <details> <summary>Release notes</summary> _Sourced from [Umbraco.Cms.Persistence.Sqlite's releases](https://github.com/umbraco/Umbraco-CMS/releases)._ ## 17.3.0 ## Upgrade Notes In 17.3 we have upgraded our dependency on `MailKit` to 4.15.1. This is a minor version update, but we found a few changes we had to make in core to accommodate changes to nullability constraints. Unless using methods of this library, or it's transitive dependency `MimeKit`, it's unlikely projects will be affected. The update is necessary though, as the version we previously depended on now has a security vulnerability raised against it. We have made a change to how we handle redirects which brings a significant performance improvement for publish time on large sites, when documents with many descendent nodes are published. If you have custom URL providers you should review this change, as there are some [very rare cases](umbraco/Umbraco-CMS#22091 (comment)) where you'll need to adjust to ensure descendent redirects are correctly handled. Note also that we now auto-generate HMAC secret key for new installs. This has been applied to make Umbraco more secure by default, but it's not been forced for upgrades. ## What's Changed Since 17.3.0-rc3 **Full Changelog**: umbraco/Umbraco-CMS@release-17.3.0-rc3...release-17.3.0 ## What's Changed Since 17.3.0-rc2 ### 📦 Dependencies * Dependencies: Update Microsoft packages to latest patch and fix HybridCache ParseFault with Redis by @AndyButland in umbraco/Umbraco-CMS#22278 ### 🐛 Bug Fixes * Examine: Fix DocumentUrlService not initialized during Examine indexing after package upgrade by @AndyButland in umbraco/Umbraco-CMS#22243 * Unattended Upgrades: Rebuild routing caches after background migrations to fix unrouteable document URLs by @AndyButland in umbraco/Umbraco-CMS#22269 * Migrations: Fix NPoco auto-select breaking re-trust FK migration by @AndyButland in umbraco/Umbraco-CMS#22270 **Full Changelog**: umbraco/Umbraco-CMS@release-17.3.0-rc2...release-17.3.0-rc3 ## What's Changed Since 17.3.0-rc1 ### 🐛 Bug Fixes * Migrations: Fix re-trust constraints migration targeting non-Umbraco tables and transaction failure (closes #22227) by @AndyButland in umbraco/Umbraco-CMS#22229 * Distributed Locking: Add ROWLOCK hint to prevent cross-row contention on umbracoLock table (closes #22113) by @AndyButland in umbraco/Umbraco-CMS#22126 * Application URLs: Prevent back office hosts being overwritten in a shared database setup (closes #16741) by @matthewcare in umbraco/Umbraco-CMS#22160 * Migrations: Fix property detection for invariant content types with culture-varying compositions (closes #22159) by @AndyButland in umbraco/Umbraco-CMS#22167 * Migrations: Fix package migrations not running after fresh install with packages (closes #22202) by @AndyButland in umbraco/Umbraco-CMS#22204 **Full Changelog**: umbraco/Umbraco-CMS@release-17.3.0-rc...release-17.3.0-rc2 ## What's Changed Since the Previous Version (17.2.2) ### 🙌 Notable Changes * Templates: Add optional Central Package Management support to UmbracoProject and UmbracoExtension templates by @NguyenThuyLan in umbraco/Umbraco-CMS#21641 * Service registration: Allow running Umbraco with different combinations of backoffice, website and delivery API (closes #21622) by @AndyButland in umbraco/Umbraco-CMS#21630 * Imaging Configuration: Auto-generate HMAC secret key for new installs by @AndyButland in umbraco/Umbraco-CMS#21976 * Migrations: Run unattended upgrades in background, add liveness/readiness health probes (closes #21987) by @AndyButland in umbraco/Umbraco-CMS#22020 ### 💥 Breaking Changes * Dependencies: Update MailKit to 4.15.1 by @AndyButland in umbraco/Umbraco-CMS#22028 ### 📦 Dependencies * Bump lodash from 4.17.21 to 4.17.23 in /tests/Umbraco.Tests.AcceptanceTest in the npm_and_yarn group across 1 directory by @dependabot[bot] in umbraco/Umbraco-CMS#21519 ... (truncated) ## 17.3.0-rc3 ## Upgrade Notes In 17.3 we have upgraded our dependency on `MailKit` to 4.15.1. This is a minor version update, but we found a few changes we had to make in core to accommodate changes to nullability constraints. Unless using methods of this library, or it's transitive dependency `MimeKit`, it's unlikely projects will be affected. The update is necessary though, as the version we previously depended on now has a security vulnerability raised against it. We have made a change to how we handle redirects which brings a significant performance improvement for publish time on large sites, when documents with many descendent nodes are published. If you have custom URL providers you should review this change, as there are some [very rare cases](umbraco/Umbraco-CMS#22091 (comment)) where you'll need to adjust to ensure descendent redirects are correctly handled. Note also that we now auto-generate HMAC secret key for new installs. This has been applied to make Umbraco more secure by default, but it's not been forced for upgrades. ## What's Changed Since 17.3.0-rc2 ### 📦 Dependencies * Dependencies: Update Microsoft packages to latest patch and fix HybridCache ParseFault with Redis by @AndyButland in umbraco/Umbraco-CMS#22278 ### 🐛 Bug Fixes * Examine: Fix DocumentUrlService not initialized during Examine indexing after package upgrade by @AndyButland in umbraco/Umbraco-CMS#22243 * Unattended Upgrades: Rebuild routing caches after background migrations to fix unrouteable document URLs by @AndyButland in umbraco/Umbraco-CMS#22269 * Migrations: Fix NPoco auto-select breaking re-trust FK migration by @AndyButland in umbraco/Umbraco-CMS#22270 **Full Changelog**: umbraco/Umbraco-CMS@release-17.3.0-rc2...release-17.3.0-rc3 ## What's Changed Since 17.3.0-rc1 ### 🐛 Bug Fixes * Migrations: Fix re-trust constraints migration targeting non-Umbraco tables and transaction failure (closes #22227) by @AndyButland in umbraco/Umbraco-CMS#22229 * Distributed Locking: Add ROWLOCK hint to prevent cross-row contention on umbracoLock table (closes #22113) by @AndyButland in umbraco/Umbraco-CMS#22126 * Application URLs: Prevent back office hosts being overwritten in a shared database setup (closes #16741) by @matthewcare in umbraco/Umbraco-CMS#22160 * Migrations: Fix property detection for invariant content types with culture-varying compositions (closes #22159) by @AndyButland in umbraco/Umbraco-CMS#22167 * Migrations: Fix package migrations not running after fresh install with packages (closes #22202) by @AndyButland in umbraco/Umbraco-CMS#22204 **Full Changelog**: umbraco/Umbraco-CMS@release-17.3.0-rc...release-17.3.0-rc2 ## What's Changed Since the Previous Version (17.2.2) ### 🙌 Notable Changes * Templates: Add optional Central Package Management support to UmbracoProject and UmbracoExtension templates by @NguyenThuyLan in umbraco/Umbraco-CMS#21641 * Service registration: Allow running Umbraco with different combinations of backoffice, website and delivery API (closes #21622) by @AndyButland in umbraco/Umbraco-CMS#21630 * Imaging Configuration: Auto-generate HMAC secret key for new installs by @AndyButland in umbraco/Umbraco-CMS#21976 * Migrations: Run unattended upgrades in background, add liveness/readiness health probes (closes #21987) by @AndyButland in umbraco/Umbraco-CMS#22020 ### 💥 Breaking Changes * Dependencies: Update MailKit to 4.15.1 by @AndyButland in umbraco/Umbraco-CMS#22028 ### 📦 Dependencies * Bump lodash from 4.17.21 to 4.17.23 in /tests/Umbraco.Tests.AcceptanceTest in the npm_and_yarn group across 1 directory by @dependabot[bot] in umbraco/Umbraco-CMS#21519 * Bump the npm_and_yarn group across 2 directories with 2 updates by @dependabot[bot] in umbraco/Umbraco-CMS#21754 * Bump qs from 6.14.1 to 6.14.2 in /src/Umbraco.Web.UI.Client in the npm_and_yarn group across 1 directory by @dependabot[bot] in umbraco/Umbraco-CMS#21755 * Dependencies: Bumps @umbraco-ui/uui from 1.17.0 to 1.17.1 by @iOvergaard in umbraco/Umbraco-CMS#22029 * Dependencies: Updates @umbraco-ui/uui to 1.17.2 to fix multiple folder drag-and-drop failing (closes #21837) by @iOvergaard in umbraco/Umbraco-CMS#21886 ... (truncated) ## 17.3.0-rc2 ## Upgrade Notes In 17.3 we have upgraded our dependency on `MailKit` to 4.15.1. This is a minor version update, but we found a few changes we had to make in core to accommodate changes to nullability constraints. Unless using methods of this library, or it's transitive dependency `MimeKit`, it's unlikely projects will be affected. The update is necessary though, as the version we previously depended on now has a security vulnerability raised against it. We have made a change to how we handle redirects which brings a significant performance improvement for publish time on large sites, when documents with many descendent nodes are published. If you have custom URL providers you should review this change, as there are some [very rare cases](umbraco/Umbraco-CMS#22091 (comment)) where you'll need to adjust to ensure descendent redirects are correctly handled. Note also that we now auto-generate HMAC secret key for new installs. This has been applied to make Umbraco more secure by default, but it's not been forced for upgrades. ## What's Changed Since 17.3.0-rc1 ### 🐛 Bug Fixes * Migrations: Fix re-trust constraints migration targeting non-Umbraco tables and transaction failure (closes #22227) by @AndyButland in umbraco/Umbraco-CMS#22229 * Distributed Locking: Add ROWLOCK hint to prevent cross-row contention on umbracoLock table (closes #22113) by @AndyButland in umbraco/Umbraco-CMS#22126 * Application URLs: Prevent back office hosts being overwritten in a shared database setup (closes #16741) by @matthewcare in umbraco/Umbraco-CMS#22160 * Migrations: Fix property detection for invariant content types with culture-varying compositions (closes #22159) by @AndyButland in umbraco/Umbraco-CMS#22167 * Migrations: Fix package migrations not running after fresh install with packages (closes #22202) by @AndyButland in umbraco/Umbraco-CMS#22204 **Full Changelog**: umbraco/Umbraco-CMS@release-17.3.0-rc...release-17.3.0-rc2 ## What's Changed Since the Previous Version (17.2.2) ### 🙌 Notable Changes * Templates: Add optional Central Package Management support to UmbracoProject and UmbracoExtension templates by @NguyenThuyLan in umbraco/Umbraco-CMS#21641 * Service registration: Allow running Umbraco with different combinations of backoffice, website and delivery API (closes #21622) by @AndyButland in umbraco/Umbraco-CMS#21630 * Imaging Configuration: Auto-generate HMAC secret key for new installs by @AndyButland in umbraco/Umbraco-CMS#21976 * Migrations: Run unattended upgrades in background, add liveness/readiness health probes (closes #21987) by @AndyButland in umbraco/Umbraco-CMS#22020 ### 💥 Breaking Changes * Dependencies: Update MailKit to 4.15.1 by @AndyButland in umbraco/Umbraco-CMS#22028 ### 📦 Dependencies * Bump lodash from 4.17.21 to 4.17.23 in /tests/Umbraco.Tests.AcceptanceTest in the npm_and_yarn group across 1 directory by @dependabot[bot] in umbraco/Umbraco-CMS#21519 * Bump the npm_and_yarn group across 2 directories with 2 updates by @dependabot[bot] in umbraco/Umbraco-CMS#21754 * Bump qs from 6.14.1 to 6.14.2 in /src/Umbraco.Web.UI.Client in the npm_and_yarn group across 1 directory by @dependabot[bot] in umbraco/Umbraco-CMS#21755 * Dependencies: Bumps @umbraco-ui/uui from 1.17.0 to 1.17.1 by @iOvergaard in umbraco/Umbraco-CMS#22029 * Dependencies: Updates @umbraco-ui/uui to 1.17.2 to fix multiple folder drag-and-drop failing (closes #21837) by @iOvergaard in umbraco/Umbraco-CMS#21886 * Backoffice: Update vite from 7.1.11 to 7.3.1 by @iOvergaard in umbraco/Umbraco-CMS#22065 * Dependencies: Update server-side dependencies to latest patch or minor releases by @AndyButland in umbraco/Umbraco-CMS#21860 ### 🚤 Performance * Performance: Implement key-based caching for data type and template repositories by @AndyButland in umbraco/Umbraco-CMS#21280 * Management API: Optimize collection view performance by eliminating N+1 patterns by @AndyButland in umbraco/Umbraco-CMS#21684 * Performance: Optimize handling of content type updates by @kjac in umbraco/Umbraco-CMS#21910 * Backoffice Performance: Add inflight request deduplication to item data request managers by @madsrasmussen in umbraco/Umbraco-CMS#21767 * URL and Alias Caches: Optimize for invariant documents by @AndyButland in umbraco/Umbraco-CMS#21558 * Custom Views: Prevent re-rendering Block Views and Properties by @rickbutterfield in umbraco/Umbraco-CMS#21186 * Core: Minimize await to a single JS cycle (refactor #21186) by @nielslyngsoe in umbraco/Umbraco-CMS#22074 * Auth: Skip /token refresh when access token is still valid by @iOvergaard in umbraco/Umbraco-CMS#22087 * Memory Management: Dispose `IDisposable` resources correctly in four internal classes by @AndyButland in umbraco/Umbraco-CMS#22014 * Redirect Tracking: Fix segment change detection and optimise descendant traversal (closes #22082) by @AndyButland in umbraco/Umbraco-CMS#22091 ... (truncated) ## 17.3.0-rc ## Upgrade Notes In 17.3 we have upgraded our dependency on `MailKit` to 4.15.1. This is a minor version update, but we found a few changes we had to make in core to accommodate changes to nullability constraints. Unless using methods of this library, or it's transitive dependency `MimeKit`, it's unlikely projects will be affected. The update is necessary though, as the version we previously depended on now has a security vulnerability raised against it. We have made a change to how we handle redirects which brings a significant performance improvement for publish time on large sites, when documents with many descendent nodes are published. If you have custom URL providers you should review this change, as there are some [very rare cases](umbraco/Umbraco-CMS#22091 (comment)) where you'll need to adjust to ensure descendent redirects are correctly handled. Note also that we now auto-generate HMAC secret key for new installs. This has been applied to make Umbraco more secure by default, but it's not been forced for upgrades. ## What's Changed ### 🙌 Notable Changes * Templates: Add optional Central Package Management support to UmbracoProject and UmbracoExtension templates by @NguyenThuyLan in umbraco/Umbraco-CMS#21641 * Service registration: Allow running Umbraco with different combinations of backoffice, website and delivery API (closes #21622) by @AndyButland in umbraco/Umbraco-CMS#21630 * Imaging Configuration: Auto-generate HMAC secret key for new installs by @AndyButland in umbraco/Umbraco-CMS#21976 * Migrations: Run unattended upgrades in background, add liveness/readiness health probes (closes #21987) by @AndyButland in umbraco/Umbraco-CMS#22020 ### 💥 Breaking Changes * Dependencies: Update MailKit to 4.15.1 by @AndyButland in umbraco/Umbraco-CMS#22028 ### 📦 Dependencies * Bump lodash from 4.17.21 to 4.17.23 in /tests/Umbraco.Tests.AcceptanceTest in the npm_and_yarn group across 1 directory by @dependabot[bot] in umbraco/Umbraco-CMS#21519 * Bump the npm_and_yarn group across 2 directories with 2 updates by @dependabot[bot] in umbraco/Umbraco-CMS#21754 * Bump qs from 6.14.1 to 6.14.2 in /src/Umbraco.Web.UI.Client in the npm_and_yarn group across 1 directory by @dependabot[bot] in umbraco/Umbraco-CMS#21755 * Dependencies: Bumps @umbraco-ui/uui from 1.17.0 to 1.17.1 by @iOvergaard in umbraco/Umbraco-CMS#22029 * Dependencies: Updates @umbraco-ui/uui to 1.17.2 to fix multiple folder drag-and-drop failing (closes #21837) by @iOvergaard in umbraco/Umbraco-CMS#21886 * Backoffice: Update vite from 7.1.11 to 7.3.1 by @iOvergaard in umbraco/Umbraco-CMS#22065 * Dependencies: Update server-side dependencies to latest patch or minor releases by @AndyButland in umbraco/Umbraco-CMS#21860 ### 🚤 Performance * Performance: Implement key-based caching for data type and template repositories by @AndyButland in umbraco/Umbraco-CMS#21280 * Management API: Optimize collection view performance by eliminating N+1 patterns by @AndyButland in umbraco/Umbraco-CMS#21684 * Performance: Optimize handling of content type updates by @kjac in umbraco/Umbraco-CMS#21910 * Backoffice Performance: Add inflight request deduplication to item data request managers by @madsrasmussen in umbraco/Umbraco-CMS#21767 * URL and Alias Caches: Optimize for invariant documents by @AndyButland in umbraco/Umbraco-CMS#21558 * Custom Views: Prevent re-rendering Block Views and Properties by @rickbutterfield in umbraco/Umbraco-CMS#21186 * Core: Minimize await to a single JS cycle (refactor #21186) by @nielslyngsoe in umbraco/Umbraco-CMS#22074 * Auth: Skip /token refresh when access token is still valid by @iOvergaard in umbraco/Umbraco-CMS#22087 * Memory Management: Dispose `IDisposable` resources correctly in four internal classes by @AndyButland in umbraco/Umbraco-CMS#22014 * Redirect Tracking: Fix segment change detection and optimise descendant traversal (closes #22082) by @AndyButland in umbraco/Umbraco-CMS#22091 ### 🌈 Accessibility Improvements * Entity Actions: Adds a descriptive title to the first action so you know what it does by @iOvergaard in umbraco/Umbraco-CMS#21739 * Search field: Added aria-label and name to search input for accessibility (closes #21938) by @andreaslborg in umbraco/Umbraco-CMS#21962 * List view: Added labels entity bulk action buttons by @andreaslborg in umbraco/Umbraco-CMS#21964 * Accessibility: Add title attributes to buttons in block list entry and property editor UI by @manutdkid77 in umbraco/Umbraco-CMS#21842 * Accessibility: Add tooltips to block grid entry actions by @manutdkid77 in umbraco/Umbraco-CMS#21958 * Accessibility: Added `title` attribute for icon in content types by @TechPdo in umbraco/Umbraco-CMS#21956 ### 🚀 New Features ... (truncated) Commits viewable in [compare view](umbraco/Umbraco-CMS@release-17.2.2...release-17.3.0). </details> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Alexander Seeliger <alexsee@users.noreply.github.com>
Description
Follows investigation into: #22113
This PR adds a
ROWLOCKtable hint to SQL Server distributed locking queries (both NPoco and EF Core) to prevent SQL Server from choosing page-level lock granularity on the smallumbracoLocktable.Analysis
The
umbracoLocktable has ~18 rows which all fit on a single 8KB SQL Server data page. The existing locking SQL usesWITH (REPEATABLEREAD)but noROWLOCKhint, leaving lock granularity up to SQL Server's query optimizer.From what I understand, it's possible for SQL Server to choose a page or table lock here, which could lead problems similar to that described in the reported issue.
For example, when a long-running content operation (e.g.,
EmptyRecycleBin) holds a lock onContentTree(-333), SQL Server may use a page-level lock that also covers theDistributedJobsrow (-347). Other servers pollingTryTakeRunnableAsyncevery 5 seconds then fail to acquireWriteLock(-347), producingDistributedWriteLockTimeoutException("Failed to acquire write lock for id: -347").Adding
ROWLOCKensures SQL Server locks only the specific row being accessed, so locks on different IDs never contend with each other regardless of table size or optimizer decisions.Testing
I've used some local integration tests to verify that if a page lock is explicitly requested, then we do see the contention between the row locks. But as this isn't deterministic it's not possible to demonstrate conclusively that this problem occurs and is resolved by this update.
My suggestion is that I can't see any harm in doing this. We definitely want row level locking on this table, as that's the point of being able to lock certain aggregate roots and not others. So being explicit could be useful here in some where lock escalation to a page or table level could otherwise occur.